{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.5 正则化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在深度学习中，常用的正则化技术包括 L2正则化、L1正则化和权重衰减。\n",
    "\n",
    "L2正则化可以使得模型的参数值趋于平稳，防止过拟合。但是，L2正则化也有一些缺点，例如使得训练变慢、对稀疏数据无效等。因此，在使用 L2正则化时，我们需要根据实际情况选择合适的正则化系数 $\\lambda$ ，使得模型能够得到较好的泛化能力。\n",
    "\n",
    "L1正则化是另一种常用的正则化技术，它可以有效防止稀疏数据的过拟合。但是，L1正则化的惩罚项是参数的绝对值和，会使得模型的参数分布呈均匀分布。在使用 L1正则化时，我们也需要根据实际情况选择合适的正则化系数 $\\lambda$ ，使得模型能够得到较好的泛化能力。\n",
    "\n",
    "权重衰减是也是一种常用的正则化技术，它可以有效防止稀疏数据的过拟合。权重衰减的惩罚项是参数的平方，会使得模型的参数取值趋于较小。在使用权重衰减时，我们也需要根据实际情况选择合适的正则化系数 $\\lambda$ ，使得模型能够得到较好的泛化能力。\n",
    "\n",
    "在深度学习中，L2正则化是三种正则化技术中应用最广泛的。这是因为 L2正则化可以使得模型的参数值趋于平稳，防止过拟合。此外，L2正则化的计算复杂度也比较低，因此在训练模型时也不会太慢。我们这里重点介绍这种方法。\n",
    "\n",
    "**梗直哥提示：对正则化的原理感兴趣，希望深入了解的同学，可以点击选修[“解一卷而众篇明”之机器学习核心概念精讲](https://aay.xet.tech/s/2TMAev)，用具体生动的例子，把正则化的前因后果，数学推导，几何意义，与MAP最大后验的关系等等各种知识给你讲个通透。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5.1 L2正则化\n",
    "\n",
    "在深度学习中，模型的复杂度往往随着层数的增加而增加。当模型太复杂时，它会记住训练数据中的细节，并且在测试数据上表现很差。这种现象被称为过拟合。为了避免过拟合，我们需要让模型尽量简单，这就是正则化的目的。\n",
    "\n",
    "L2正则化通过给模型的损失函数添加一个模型参数的平方和的惩罚项来实现正则化。具体来说，我们可以把损失函数写成如下的形式：\n",
    "\n",
    "$$Loss = Loss_{original} + \\lambda \\sum_{i=1}^n w_i^2$$\n",
    "\n",
    "其中 $Loss_{original}$ 是原始的损失函数，$\\lambda$ 是正则化系数，$w_i$ 是模型的参数，$n$ 是参数的数量。这个惩罚项的作用是给模型的参数加上一个惩罚，如果参数的值过大，则惩罚值也会很大，从而使得模型的参数值趋于平稳。\n",
    "\n",
    "L2正则化有好几个重要优点：我们刚说过，它会使得模型的参数值趋于平稳。这意味着参数值不会过大，也不会过小。这样可以避免出现梯度爆炸和梯度消失的现象。它还有一个重要的优点：它可以防止过拟合。因为 L2正则化会惩罚模型的参数值，所以当模型的参数值较大时，损失函数的值也会较大。这样可以防止模型过拟合训练数据。当然，L2正则化也有一些缺点。例如，它会使得模型的训练变慢，因为损失函数中多了一个惩罚项，需要计算。此外，L2正则化也不能有效防止稀疏数据中的过拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5.2 代码示例\n",
    "\n",
    "下面是一个示例，展示了在使用和不使用 L2 正则化的情况下，训练误差和测试误差的变化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAyfklEQVR4nO3deXhV9bn3//ediQyEMIQp2YmAyDwkIaLWoShSQZEZROo5p56BPs859nhaterpbH/nV/0dH2t7PXawHq3tOVUxjIoyOFAnVBISIAOzaPZOIAFkHjLdvz++C9ilEAJJ2Nl736/r2pfstdde+a6r6fpkrfs7iKpijDEm+sSEugHGGGNCwwLAGGOilAWAMcZEKQsAY4yJUhYAxhgTpeJC3YCLkZ6erv369Qt1M4wxJmwUFRXtVdWe5/osrAKgX79+FBYWhroZxhgTNkTk8/N9Zo+AjDEmSlkAGGNMlLIAMMaYKBVWNQBjjLlY9fX1+P1+Tpw4EeqmtKvExER8Ph/x8fEt/o4FgDEmovn9flJTU+nXrx8iEurmtAtVZd++ffj9fvr379/i79kjIGNMRDtx4gQ9evSI2Is/gIjQo0ePi77LsQAwxkS8SL74n3Ip5xgVAfDLt7fx7uYaGpts6mtjjDkl4gPg6MkG/rD2c+79/Tquf/wdnly5hS/2HQt1s4wxUeLAgQP86le/uujv3X777Rw4cKDtGxSkRQEgIhNFZIuIbBeRR5rZb6aIqIjke+/jReRFEdkkIhUi8qi3fbCIlAS9DonIv7XJGZ0lpVMcHz1yC7+5J48hfVP51Zrt3PSf7zLvdx+ztCTAifrG9vixxhgDnD8AGhoamv3eG2+8QdeuXdupVc4FewGJSCzwDDAB8APrRGSZqpaftV8qcD/wSdDm2UAnVR0pIslAuYi8pKpbgJyg4weAxW1wPueUEBfDxBF9mTiiL9UHj1NQ6OeVwkruf7mEtKR4puVkMOfqLIZnpLVXE4wxUeqRRx5hx44d5OTkEB8fT2JiIt26dWPz5s1s3bqVadOmUVlZyYkTJ7j//vuZP38+cGbqmyNHjjBp0iRuuOEGPvroIzIzM1m6dClJSUmtbltLuoGOBbar6k4AEXkZmAqUn7XfT4EngIeCtimQIiJxQBJQBxw663vjgR2qet75KtpS37QkvjX+Kv7l5oGs3bmPV9ZV8tK6Sl5c+zkjMrtwV34WU3IySUtqeV9aY0x4+MlrZZRXnX0Jap1hGV340Z3Dz/v5448/TmlpKSUlJaxZs4Y77riD0tLS0901n3/+ebp3787x48e5+uqrmTlzJj169PiLY2zbto2XXnqJ3/3ud8yZM4eFCxdyzz33tLrtLQmATKAy6L0fuCZ4BxHJA7JUdbmIBAdAAS4sqoFk4Nuquv+s488FXrrYhrdWTIxw/cB0rh+YzoFjdSwtqeLldZX8YGkZ/8/yCiaN6MOcq7O4tn8PYmIivweBMebyGDt27F/01f/lL3/J4sXuAUhlZSXbtm37qwDo378/OTk5AIwZM4Zdu3a1SVtaPRBMRGKAp4BvnOPjsUAjkAF0A94XkbeC7iYSgCnAo80cfz4wHyA7O7u1zT2nrskJ/N1X+vG3111BaeAQrxR+wdKSKpaUVJHdPZk5+T5mjcmiT1piu/x8Y8zl0dxf6pdLSkrK6X+vWbOGt956i7Vr15KcnMy4cePO2Ze/U6dOp/8dGxvL8ePH26QtLQmAAJAV9N7nbTslFRgBrPH6ofYBlonIFGAesEJV64EaEfkQyAd2et+dBKxX1T3n++Gq+izwLEB+fn679uMUEUb60hjpG8n3bh/GirJqXllXyZOrtvLU6q2MG9yLOfk+bhnSm4S4iO9AZYxpA6mpqRw+fPicnx08eJBu3bqRnJzM5s2b+fjjjy9r21oSAOuAq0SkP+7CPxd3YQdAVQ8C6afei8ga4EFVLRSR8cAtwB9FJAW4Fng66Nh3E4LHPy2RlBDL9Fwf03N97Np7lFeLKiko8vO//ruGHikJzMjL5K6rsxjYKzXUTTXGdGA9evTg+uuvZ8SIESQlJdG7d+/Tn02cOJHf/OY3DB06lMGDB3Pttdde1raJ6oX/qBaR23EX7ljgeVX9DxF5DChU1WVn7buGMwHQGXgBGAYI8IKq/qe3XwrwBTDAC5ELys/P11AuCNPQ2MR722pZsM7PWxV7aGhScrO7cld+FpNHZ9C5k02tZExHU1FRwdChQ0PdjMviXOcqIkWqmn+u/VsUAB1FqAMg2N4jJ1m8PsArhZVsrzlCckIst4/sy11XZ5F/RbeoGHpuTDiwADh/ANifrJcovXMn/ummAfzjjf0prjzAgnWVvLahioIiPwPSU5idn8XMvEx6dbHCsTGmY7IAaCURIS+7G3nZ3fjhncNYvrGaVwv9PLFiM0+u2sLNg3syJz+Lm4f0Ij7WCsfGmI7DAqANJSfEMTs/i9n5WeysPcKrRX4WFvl5q6KG9M6dmJmXyex8nxWOjTEdggVAOxnQszMPTxzCAxMG8eettbyyrpL/+uAzfvveTvKyuzLHCsfGmBCzq087i4uNYfzQ3owf2pvawydZUuwKx48s2sRPXivnjlF9mZOfxdX9rHBsjLm87KH0ZdQz1RWOV3/7Jhb+768wNSeDNzdVM+e3a7nl//yZX63Zzp5Dkb1uqTHR5lKngwZ4+umnOXas/aavtwAIARFhzBXdeHzmKNZ9/1aenD2anqmd+P9WbOG6n73N3/9+HStKq6lraAp1U40xrdSRA8AeAYVYckIcs8b4mDXGx2d7j/JqoRtx/M5mN+J4em4ms/OzGNzHCsfGhKPg6aAnTJhAr169WLBgASdPnmT69On85Cc/4ejRo8yZMwe/309jYyM/+MEP2LNnD1VVVdx8882kp6fz7rvvtnnbLAA6kP7pKXx34hC+M2EQ72/by4LCSl5cu4vnPviM0VldmZPv487RGXRJtKmqjbkkbz4Cuze17TH7jIRJj5/34+DpoFetWkVBQQGffvopqsqUKVN47733qK2tJSMjg+XLlwNujqC0tDSeeuop3n33XdLT0897/NawAOiA4mJjuHlIL24e0ot9R06ypKSKBesq+d7iUh57rZzbR/Zl9hgf1w6wqaqNCSerVq1i1apV5ObmAnDkyBG2bdvGjTfeyAMPPMDDDz/M5MmTufHGGy9LeywAOrgenTvxDzf05++v78dG/0FeLapkaUkVi4sD+LolMXtMFrPyfWR2bf3qQMZEvGb+Ur8cVJVHH32Ub37zm3/12fr163njjTf4/ve/z/jx4/nhD3/Y7u2xAAgTIsLorK6MzurK9+8Yxsqy3SworOTnb23l6be3csPAdGbnZ/G1Yb1JjI8NdXONMZ7g6aBvu+02fvCDH/D1r3+dzp07EwgEiI+Pp6Ghge7du3PPPffQtWtXnnvuub/4rj0CMqclxscyNSeTqTmZVO4/RkGRn4IiP//6UjFdEuOYmpPJnPwsRmR2sbEFxoRY8HTQkyZNYt68eVx33XUAdO7cmf/+7/9m+/btPPTQQ8TExBAfH8+vf/1rAObPn8/EiRPJyMholyKwzQYaIZqalLU797GgsJIVpbs52dDEkD6pzM7PYlpOBj06d7rwQYyJQDYbqM0GGvGC1zg+eLye1zZUsaCwkp++Xs7jb1Ywfkhv5lzt46arehJnk9IZY7AAiEhpSfHcc+0V3HPtFWzZfZhXCytZXBxgRdlueqV2Ykaej9n5Pq7s2TnUTTXGhJAFQIQb3CeV708exncnDuGdzTUUFFXyu/d38ps/72DMFd2YPcbHHaP6kmpjC0wEU9WIr4ddyuN8qwFEoZpDJ1hcHODVIj/ba46QFB/LpBF9mJ2fxTX9u9vYAhNRPvvsM1JTU+nRo0fEhoCqsm/fPg4fPkz//v3/4rNWLwkpIhOBX+DWBH5OVc/ZmVZEZgIFwNXemsDxwHNAHu5u4w+q+jNv367eZyMABf5eVdc21w4LgLalqpRUHmBBoZ/XN1Rx+GQDWd2TmJWXxcwxmfi6JYe6ica0Wn19PX6/nxMnInuixcTERHw+H/Hxf3k336oAEJFYYCswAfAD64C7VbX8rP1SgeVAAnCfFwDzgCmqOldEkoFyYJyq7hKRF4H3VfU5EUkAklX1QHNtsQBoP8frGllZtptXiyr5cPs+ROArV/Zg9pgsJo7oY2MLjAlTre0FNBbYrqo7vYO9DEzFXcyD/RR4AngoaJsCKSISByQBdcAhEUkDbgK+AaCqdd5nJkSSEmKZlpvJtNxM/F8eY2FRgIL1lfzbKyWkLolj8ugMZuf7yM3qGrG30cZEm5b0B8wEKoPe+71tp4lIHpClqsvP+m4BcBSoBr4AnlTV/UB/oBZ4QUSKReQ5EUk51w8XkfkiUigihbW1tS06KdM6vm7J3H/rVfz5wZt56Z+uZcKw3iwu9jPjVx8x4efv8Zs/76DG1i0wJuy1ukO4iMQATwEPnOPjsUAjkIG76D8gIgNwdx55wK9VNRcXEo+c6/iq+qyq5qtqfs+ePVvbXHMRYmKE667swVN35bDue7fy+IyRpCXF8/ibm7nu8Xf4+9+v481Ntm6BMeGqJY+AAkBW0Huft+2UVFwhd433aKAPsExEpgDzgBWqWg/UiMiHQD7wHuBX1U+8YxRwngAwHUNqYjxzx2Yzd2w2O2uPUFDkZ+F6t25Bt+R4puZkMmuMjxGZaaFuqjGmhVpSBI7DFYHH4y7864B5qlp2nv3XAA96ReCHgSGqeq/3iGcdMFdVN4rI+8A/quoWEfkxkKKqD53rmKdYEbhjaWxS3t9Wy6tFflaX7aGusYmhfbswe4yPabmZdE9JCHUTjYl6rSoCq2qDiNwHrMR1A31eVctE5DGgUFWXNfP1Z3DP+csAAV5Q1Y3eZ98C/sfrAbQTuLflp2Q6gtgYYdzgXowb3IsDx+pYtqGKgiI/j71ezs/erOCWIb2YPSaLrw7uSbxNP2FMh2MDwUyb27z7EAWFfpaUBNh7pI70zp2YnpvBrDG2tKUxl1urB4J1FBYA4aW+sYk1W2opKKrk7YoaGpqUUb40Zo3xMWV0Bl2T7RGRMe3NAsCE3L4jJ1laUsWrRX4qqg+REBvDhGG9mTXGx41XpdsMpca0EwsA06GUVR2koMjP0pIq9h+to1dqJ6bnZTIrz8dVve0RkTFtyQLAdEh1DU3eDKV+3t1SQ2OTMjqrq3tENCqDtGSbodSY1rIAMB1e7eGTLC0J8Gqhny17DpMQF8PXTj8i6kmszVBqzCWxADBhQ1UpqzpEQZHrRXTgWD29u7hFbGbm+RjYyxaxMeZiWACYsHSyoZF3N9fwaqGfNVtraWxScrPdI6LJozJIS7JHRMZciAWACXv2iMiYS2MBYCLG+R4RTc/1MWtMJgN7WS8iY4JZAJiIdOoRketFVGu9iIw5BwsAE/HO9Yjo9ECzgTbQzEQvCwATNYIfES0tCfDlsXo30CzXTVdtA81MtLEAMFHpnAPNfGnMtLmITBSxADBRb++RkywpDlBQ5Gfz7sMkxMYwfmgvZo3xcdMgm67aRC4LAGOClFUdZGFRgKUlAfYdrSO9c8LpFc2G9u0S6uYZ06YsAIw5h+Dpqt/ZXEN9ozKsbxdmjvExNSeD9M6dQt1EY1rNAsCYC9h/tI7XvBXNNgUOEhcjjBvck5l5Pm4Z2otOcbGhbqIxl6TVASAiE4Ff4JaEfE5VHz/PfjNxC7xf7a0JHA88B+Thlp/8g6r+zNt3F3AYaAQaztfAYBYA5nLYuucwC4v8LC4OUHP4JF2T45kyOoOZeT5G+dIQsVHHJny0KgBEJBa3KPwEwI9b2P1uVS0/a79UYDmQANznBcA8YIqqzhWRZKAcGKequ7wAyFfVvS09EQsAczk1NDbxwfa9LFwfYFXZbk42NDGwV2dm5vmYnptJn7TEUDfRmAtq1aLwwFhgu6ru9A72MjAVdzEP9lPgCeChoG0KpIhIHJAE1AGHLq75xoRGXGzM6UXvDx6v541N1Sws8vPEis3858rNXD8wnVljfHxtWB+SEuwRkQk/LQmATKAy6L0fuCZ4BxHJA7JUdbmIBAdAAS4sqoFk4Nuqut/7TIFVIqLAb1X12Us8B2PaXVpSPHePzebusdl8tvcoi9b7WbQ+wP0vl9C5Uxx3jOzLzDE+ru7XzR4RmbDRkgBolojEAE8B3zjHx2Nxz/gzgG7A+yLylnc3cYOqBkSkF7BaRDar6nvnOP58YD5AdnZ2a5trTKv1T0/hga8N5tu3DuKTz/azcL2f1zZW8UphJVndk5iR69YuyO6RHOqmGtOsltQArgN+rKq3ee8fBQgq5qYBO4Aj3lf6APuBKcC9wMeq+kdv3+eBFaq64Kyf8WPgiKo+2VxbrAZgOqpjdQ2sKN3NwvV+PtqxD1UY2687M8dkMmlkX7ok2sR0JjRaWwSOwxWBxwMBXBF4nqqWnWf/NcCDXhH4YWCIqt4rIined+fiAiNGVQ9721cDj6nqiubaYgFgwkHVgeMsLg6wcL2fnbVH6RQXw23D+zAjL9PWLjCXXauKwKraICL3AStx3UCfV9UyEXkMKFTVZc18/RngBREpAwR4QVU3isgAYLH3rDQO+NOFLv7GhIuMrkn8y80D+edxV7LBf5CFRX6Wbahi2Yaq0xPTzcjzMbiPTUxnQssGghlzGZxsaOSdihoWrg+wZksNDU3KiMwuzMh1o4572Khj005sJLAxHcjeIyd5bUMVC9f7KQ0cslHHpl1ZABjTQW3ZfZhF68+MOk5LiufO0X2ZkecjN6urdSk1rWYBYEwH19ikfLh9LwvX+1lZtpsT9U0MSE9hRl4m03Iz8XWzLqXm0lgAGBNGDp+o583S3Sws8vPJZ27c5LUDujMjz8ftI/vSuVOrh++YKGIBYEyYqtx/jCXFARYVB/hs71ES42OYOLwPM/J8XD8w3bqUmguyADAmzKkqxZUHWFjk57UNVRw60UDvLp2YlpPJzDE+Btlax+Y8LACMiSDNdSmdYgvZmLNYABgTofYdOcmyDVUsWh9gU+AgsTHCuEE9mZHnY/zQXiTGW5fSaGcBYEwU2LrnMIvWB1hSHGD3oROkJsYxeVQGM/Iyyb/CZimNVhYAxkSRxiZl7Y59LFrv583S3RyvbyS7e7I3BUUmV/RICXUTzWVkAWBMlDp6soGVZX85S+mYK7oxIy+TySMzSEu2WUojnQWAMYbqg8dZUlzFovV+ttUcISE2hvFDezEjz8dXB/UkIS4m1E007cACwBhzmqpSGjjEomI/y0qq2He0ju4pCdw5yk1BYQvfRxYLAGPMOdU3NvHe1loWFQdYXb6HuoYmBvRMYWaej2m5mWR2TQp1E00rWQAYYy7o4PF63txUzaLiAJ96U1Bc0787M/N8TBrZh1Rb1SwsWQAYYy7KqSkoFhcH2LnXrWo2YVjv06uaxcdavSBcWAAYYy6JqlJSeYDFxQGWbajiwLF6eqQkcOfoDGbm+RiR2cXqBR2cBYAxptXqGppYs6WGxcUB3q6ooa6xiYG9OjM9N9PqBR1YqwNARCYCv8CtCfycqj5+nv1mAgXA1d6i8PHAc0Aebu3fP6jqz4L2jwUKgYCqTr5QOywAjOkYDh6rZ/mmahYX+1m360vA1Qtm5GUyaWRfuli9oMNoVQB4F+mtwATAD6wD7lbV8rP2SwWWAwnAfV4AzAOmqOpcEUkGyoFxqrrL+853gHygiwWAMeGpcv8xFnv1gs+8esGtw3ozIzeTmwZZvSDUmguAlqwsMRbYrqo7vYO9DEzFXcyD/RR4AngoaJsCKSISByQBdcAh7zg+4A7gP4DvtPhsjDEdSlb3ZP51/FV865aBp+sFr22oYvnG6tP1gum5mTa+oANqSQBkApVB7/3ANcE7iEgekKWqy0UkOAAKcGFRDSQD31bV/d5nTwPfBZqdyFxE5gPzAbKzs1vQXGNMKIgIudndyM3uxvfvGMaft9aypDjAnz79gt9/tIsBPVOYkZvJ1JxMsrrbEpcdQavXlhORGOAp4Bvn+Hgs0AhkAN2A90XkLWAYUKOqRSIyrrnjq+qzwLPgHgG1tr3GmPaX4HUbnTCs91+ML3hy1VaeXLWVsf27Mz03k9tH9iUtyeoFodKSGsB1wI9V9Tbv/aMAp4q5IpIG7ACOeF/pA+wHpgD3Ah+r6h+9fZ8HVgC5wN8ADUAi0AVYpKr3NNcWqwEYE94q9x9jaYlb4nJn7VES4mK4dWgvpuVkMm5wL5uPqB20tggchysCjwcCuCLwPFUtO8/+a4AHvSLww8AQVb1XRFK8785V1Y1B+4/z9rcisDFRQlXZ6D94ul6w72gd3ZLjmTwqg2m5meRld7V6QRtpVRFYVRtE5D5gJa4b6POqWiYijwGFqrqsma8/A7wgImWAAC8EX/yNMdFJRBid1ZXRWV353h1DeX9bLYvWB1hQWMkfP/6cK3okMy0nk+m5mfRLt/UL2kt0DARraoQYWxrPmI7u8Il63izdzZLiAGt3uvULcrO7MiM3k8mjMuiWkhDqJoad6B4JrAq/ug56D4PR82DAOIhtde3bGNPOqg4cZ9mGKhavD7Blz2HiYoRxg3sxPTfT1ju+CNEdAHVHYfWPoLQAjn8JnXvDqDkw+m7oPbx9GmqMaTOqSkX1YRYX+1m2oYo9h06S2imO20f2ZVpuJtf0705MjNULzie6A+CUhpOwbRWUvATbVkJTA/QZ5YJg5Gzo3LNtG2uMaXOn1jteXBxgRWk1R+sayUhLZGquqxcM6t3ssKKoZAFwtqN7oXQhlPwJqksgJg4GToDRc2HwJIjr1PqfYYxpV8fqGlhdvofFxQHe37aXxiZlWN8uTM/NZEpOBr27JIa6iR2CBUBzaipgw0uwcQEcrobErjBihqsX+PLBuqIZ0+HVHj7J6xurWFIcYIP/ICJw/ZXpTMvNZOKIPnTuFL11PwuAlmhqhJ1rYMPLUPEaNByH7le6R0Sj74KuNg2FMeFgR+0RlhYHWFwSoHL/cRLjY5gwrA/TczOicjEbC4CLdeIQVCxz9YLPP3Db+t3owmDYFOhkzxmN6ehUlaLPv2RJSYDXN1Zz4Fg93VMSuHOUKx7nZEXHYDMLgNb48nP3eGjDS7B/B8QlwdA7Xb1gwDgbX2BMGKhraDo9Od3qij3UNTTRr0cy03IzmZYT2YPNLADagir417nCcdkiOHEQUvu6HkSj73bjDIwxHd6hE/Ws2LSbJSVnBpvlZHVlem4mk0f1pUfnyOoEYgHQ1upPuK6kJS/B9tVndSmdBZ17hbqFxpgWqD54nGUlVSwpqaKi+hCxMcJNV7ni8deG9SEpIfzv8C0A2tORWteldMOfoHoDSCwMvPVMl9J4WyfVmHCwZfdhFhcHWFYSoOrgCVISYrlteB+m5WbylSt7EBemxWMLgMulpsL1Itq4AA5XQacuMHyauzPIvs66lBoTBpqalE937WepVzw+fKKB9M6dmDI6g2m5GYzMDK+VzSwALremRtj1vguD8mVQfxS6XuHuCkbdBT2uDHULjTEtcKK+kTVbali0PsCaLbXUNTYxoGcK03Jc8Ti7R8df2cwCIJROHoHNr7sw2LkGUMi6xgXBiBmQ1C3ULTTGtMDBY/W8UVrN4uIAn37mVrbNy3bF4ztGZdC9g85UagHQURwMwKZXXRjUVkBsAgya6B4RDbwV4jrmL5Ax5i8FDnjF4+IzM5XeNKgnU3MyOlzx2AKgo1F1BeONr7hAOFoLSd1dD6LRcyEjz+oFxoSJiupDLCkJsKykiuqg4vHU3Eyu7wDFYwuAjqyxHna84waabX4DGk9C+iD3iGjUHJuCwpgw0dSkfPKZKx6/samaQ17xeLI38ni0LzTFYwuAcHH8AJQvdWHwxVq3rd+NLgyGTYXELiFtnjGmZU42NPLu5lqWlgR4e3MNdQ1N9E9P8XoSZdL/Mo48bnUAiMhE4Be4NYGfU9XHz7PfTKAAuNpbFD4eeA7Iw60//AdV/ZmIJALvAZ287QWq+qMLtSPiAyDYl7u8KShe9qagSIQhd8CouXDlLbaqmTFh4uDxelaW7mZxcYCPP3Mjj0f70piak8nk0X3pldq+01a3KgBEJBbYCkwA/MA64G5VLT9rv1RgOZAA3OcFwDxgiqrOFZFkoBwYB3wOpKjqES8kPgDuV9WPm2tLVAXAKargL4SNL7sBZ8e/hJSeMMKrF/QdbfUCY8LE7oMneG1DFUtKApRVHSJG4PqB6UzNyeS24b1JTYxv85/Z2gC4Dvixqt7mvX8UQFV/dtZ+TwOrgYeAB70AuBuYB0wH0oC1wLWquj/oe8m4APjfqvpJc22JygAI1lDnVjXb+DJsXQmNddBzyJl6QZov1C00xrTQ9prDLCmuYukGN211p7gYbh3Wm2k5mXx1UE8S4tqmeNzaAJgFTFTVf/Te/w1wjareF7RPHvA9VZ0pIms4EwDxwB+B8UAy8G1Vfdb7TixQBAwEnlHVh8/z8+cD8wGys7PHfP755y0/80h2bD+UL4ENr0Dlx4BAvxvcXcHQKVYvMCZMqCrrvzhweuTx/qN1pCXFuzWPczK4ul/r1jxu1wAQkRjgHeAbqrrrrAC4Hvhn4BtAN+B9YJKq7gw6fldgMfAtVS1tri1RfwdwPvt3nqkXfPmZm7J6yO1WLzAmzNQ3NvHB9r0sLQ6wqnwPx7w1j+/MyeCBCYMv6a6guQBoyZUhAGQFvfd5205JBUYAa7wuTn2AZSIyBff4Z4Wq1gM1IvIhkA+cDgBVPSAi7wITgWYDwJxH9wEw7hH46sNuyuoNL7spq0sXBtUL7oK+OVYvMKYDi4+N4ebBvbh5cK/Tax4vLanig217eXTS0Db/eS25A4jDFYHH4y7864B5qlp2nv3XcOYO4GFgiKreKyIp3nfnAtVAvXfxTwJWAU+o6uvNtcXuAC7CueoF6YNdEIycA12zLnwMY0yH0NDYdMkDylp1B6CqDSJyH7AS1w30eVUtE5HHgEJVXdbM158BXhCRMkCAF1R1o4iMAl706gAxwIILXfzNRYpLgKGT3ev4l1C22NUL3n7Mvfrd6ArHw6ZCYlqoW2uMaUZ7jSa2gWDRZv9nZ+Yj2r8DYju5dQtGz4Urx9t8RMZEGBsJbP6aKgTWnxlfcGyfm49oxAxXPPblW73AmAhgAWCa11gP2992k9NteQMaTrjC8qi73JrHtn6BMWHLAsC03ImDUPGae0S06wNAwTfW1QuGz4CUHqFuoTHmIlgAmEtz0O/VC15x6xfExMFVX3NhMGiirXdsTBiwADCtowq7N8GmBbCpAA5Xu/WOh01x9YIrroeY8Fww25hIZwFg2s7p9Y5fgYplUHcEumS6WsGou6D3sFC30BgTxALAtI+6Y65ovHEBbH8LtBF6j4RRs93o47TMULfQmKhnAWDa35FaN9hs4ysQKAQE+nuL2djkdMaEjAWAubz27XBBsHGBNzldohtsNuouG2xmzGVmAWBC4/RiNq+4yemCB5uNnANZY22wmTHtzALAhF5jPex4x4XB5jeg4Th06+eCYNQcSL8q1C00JiJZAJiO5eRhqHjdhcFnfwZtgoxcFwYjZkJq71C30JiIYQFgOq7Du91cRBsXQHUJSAwMGOfqBUMmQ6fOoW6hMWHNAsCEh9qtbrDZxlfgwBfeymZ3uEdEV94CsW2/YLYxkc4CwIQXVaj81CseL4bj+yG5BwyfbsVjYy6SBYAJXw11sONt94jo1Eyl3fq5kccj50DPQaFuoTEdmgWAiQwnDsHm110YnCoe9x19pnjcpW+oW2hMh9NcALRoBi8RmSgiW0Rku4g80sx+M0VERSTfex8vIi+KyCYRqRCRR73tWSLyroiUi0iZiNx/KSdmokxiF8iZB3+7BL5TAbf9v4DAqu/Bz4fBH6ZC8f+4oDDGXFBLFoWPxS0KPwHw4xZ2v1tVy8/aLxVYDiQA93mLws8DpqjqXBFJBsqBccBJoK+qrve+VwRMO/uYZ7M7AHNOtVvdtNWbFsCXu9zI40ETXfF44K0Q1ynULTQmZFq1KDwwFtiuqju9g70MTMVdzIP9FHgCeChomwIpIhIHJAF1wCFV3Q9UA6jqYRGpADLPcUxjLqznILjle3Dzv7uRx5sWQOkiKF8CiV3dwvej5kD2V2zaamOCtOT/DZlAZdB7v7ftNBHJA7JUdflZ3y0AjuIu9l8AT3oX/+Dv9gNygU/O9cNFZL6IFIpIYW1tbQuaa6KWCGRdDbf/JzywGb5e4Baw2VQAv78Dnh4Jq3/o1jYIo9qXMe2lJXcAzRKRGOAp4Bvn+Hgs0AhkAN2A90XkraC7ic7AQuDfVPWcD25V9VngWXCPgFrbXhMlYuPhqgnuVXcUtrzpisdrn4EPfwE9h8LIWa43UbcrQt1aY0KiJQEQALKC3vu8baekAiOANeL6ZvcBlonIFGAesEJV64EaEfkQyAd2ikg87uL/P6q6qNVnYsz5JKR4F/tZcHQflC+Gja/COz91r6xrXBAMnw4p6aFurTGXTUuKwHG4IvB43IV/HTBPVcvOs/8a4EGvCPwwMERV7xWRFO+7c4FNwIvAflX9t5Y21orApk19+TmUFrgwqK0AiYWB410YDL7dpqEwEaFVRWBVbRCR+4CVQCzwvKqWichjQKGqLmvm688AL4hIGSDAC6q6UURuAP4G2CQiJd6+/66qb7T8tIxppW5XwI0PwA3fgT2lrlawqQC2/RPEJ7sQGDnbTUNhaxiYCGQDwYwJ1tQElR+7bqVli+H4l5DUDYZNcz2Jsq61nkQmrNhIYGMuxalpKDYVuGko6o9BFx+MnOlGH/cebnMSmQ7PAsCY1jp5xPUk2rQAtr8N2hjUk2iWm5/ImA7IAsCYtnR0r3s8VLoQvljrtvnGuiAYPh069wpt+4wJYgFgTHs58IUbdbypAPZsOrOgzcjZbkGbxC6hbqGJchYAxlwONRXenEQFcOBziO0Eg25zYXDV1yA+MdQtNFHIAsCYy0nVzUlUWuAeEx2thU5dYOid7jFRv5sgttWD8I1pEQsAY0KlsQF2vefuCsqXQd1hSOkJw2e4OwNfvvUkMu3KAsCYjqD+BGxb5R4TbV0JjSeh6xXurmDELOg9LNQtNBHIAsCYjubEQdi83N0Z7FzjupX2Gu7GGIyYad1KTZuxADCmIztS69Yu2PQqVHqzolu3UtNGLACMCRdffu4Kx6UL3fxEEgP9v+rCYOidkJgW6haaMGMBYEw4qqlwj4hKC9xSl7Gd3PoGI2e5JS/jk0LdQhMGLACMCWeqEChyYVC2CI7sgYTOMOQOVzy+8ma3AI4x52ABYEykaGqEXR+4u4LyZXDiACR1d+sej5xl6x6bv2IBYEwkOtdspakZMGKG60mUkWtjDIwFgDER79S6x6ULYdtqaKqH7gPcI6IRM6HXkFC30ISIBYAx0eT4l1Dxmrsz2PU+aBP0HnHmzsDGGEQVCwBjotXhPd4YgwLwf+q2+a52QTB8OqT2CWnzTPtrLgBaVC0SkYkiskVEtovII83sN1NEVETyvffxIvKiiGwSkQoReTRo3+dFpEZESi/2hIwxLZTaG675Jvzjarh/I9z6YzclxYpH4P8Mgd9PhqLfw7H9oW6pCYEL3gGISCywFZgA+IF1wN2qWn7WfqnAciABuE9VC0VkHjBFVeeKSDJQDoxT1V0ichNwBPiDqo5oSWPtDsCYNlK7xdULNhXA/h0QEwdXjnc9iQZPgk6poW6haSPN3QG0ZE7ascB2Vd3pHexlYCruYh7sp8ATwENB2xRIEZE4IAmoAw4BqOp7ItLvIs7DGNNWeg6Gm/8dxj0K1Ru80ceLYNtKiEty6xiMmGnrGES4lgRAJlAZ9N4PXBO8g4jkAVmqulxEggOgABcW1UAy8G1Vvah7TRGZD8wHyM7OvpivGmMuRAQyctzr1p+4OsGmArfkZfkSSEiFoZNdGAwYZwPOIkyrV6UQkRjgKeAb5/h4LNAIZADdgPdF5K1TdxMtoarPAs+CewTU2vYaY84jJgayr3WviY+7dQxKF7oeRRte8gacTXFdS6/4CsTEhrrFppVaEgABICvovc/bdkoqMAJYI27QSR9gmYhMAeYBK1S1HqgRkQ+BfKDFAWCMCYHYOLjyFve64ynY8Y67M9i4wBWNO/dxvYhGzHC9imzAWVhqSQCsA64Skf64C/9c3IUdAFU9CKSfei8ia4AHvSLweOAW4I8ikgJcCzzdZq03xrS/uE6uMDx4khtwtnWluzMofB4++TWkZcOI6e4xUZ9RFgZh5IIBoKoNInIfsBKIBZ5X1TIReQwoVNVlzXz9GeAFESkDBHhBVTcCiMhLwDggXUT8wI9U9b9adzrGmHaVkOINKJvhLWrzhguDtc/Ah7+AHgO9MQYzbPRxGLCBYMaY1ju6DyqWudlKP3sfULfC2amw6D4g1C2MWjYS2Bhz+RzeDeVL3Z3BqRXOMvLOjD5Oywxt+6KMBYAxJjQOVLq7gtJFUF3itmVf58Jg2FRb7vIysAAwxoTevh1nwqCm3C132e9GFwZD74Tk7qFuYUSyADDGdCx7yr0wWAj7d3pTUdziwmDw7ZDYJdQtjBgWAMaYjknVTUVx6s7gYOWZtY9HzHBrHyekhLqVYa21cwEZY0z7+KupKArdXUHZYtj8OsQnn5mXaOAEm5eojdkdgDGm42lqhC/WuruC8qVwbK+bl2jI7W6MwZW3QFxCqFsZFuwRkDEmfDU2uJXNTs1LdOIAJKbBkDvdCOT+X7VJ6pphAWCMiQwNdbBzjasZbF4OJw+dmaRu+Azod4NNUncWqwEYYyJDXAIM+pp71Z+AHW+7x0QbX3WT1KX0cuMLRsyArGvdDKfmvCwAjDHhKT4RhtzhXnXH3GI2ZYuh+I+w7neQmgHDp7k7A1++TVJ3DvYIyBgTWU4ega0r3J3B9tXQWAdpWWfCICM3qsLAagDGmOh0asbSskVuTYOmBujWz81JNHwG9BkZ8WFgAWCMMcf2u8Jx2SLY+WfQRjd99akw6DU0IsPAAsAYY4Kdnr56setiqk2QPtgVj4dPh56DQ93CNmMBYIwx53Okxg02K1sMn3/E6bUMhk93r/SBoW5hq1gAGGNMSxyqdncGpYug8mO3rc/IM2EQhgvbNBcALeokKyITRWSLiGwXkUea2W+miKiI5Hvv40XkRRHZJCIVIvLoxR7TGGMumy594Zpvwj+shG+Xw20/g7gkePsx+GUu/Par8MHT8OWuULe0TVzwDkBEYoGtwATAj1sk/m5VLT9rv1RgOZAA3OctCj8PmKKqc0UkGSjHrQNc2ZJjns3uAIwxIXHgizOPiQJFbltGnndnMA26Zoe0ec1p7R3AWGC7qu5U1TrgZWDqOfb7KfAEcCJomwIpIhIHJAF1wKGLOKYxxoRe12z4yrfgn96B+zfAhMcAhdU/gKdHwu/Gw0f/Fw76Q93Si9KSAMjE/cV+it/bdpqI5AFZqrr8rO8WAEeBauAL4ElV3d+SYwYde76IFIpIYW1tbQuaa4wx7ahbP7j+fpi/Bv61BMb/CJrqYdX34OfD4bkJsPZXcDAQ4oZeWKsnyhCRGOAp4IFzfDwWaAQygP7AAyJyUVUUVX1WVfNVNb9nz56tba4xxrSd7v3hxu/AN9+Db62H8T+EhuOw8lH4+TD4r9vg41/DoapQt/ScWjIXUADICnrv87adkgqMANaIG0TRB1gmIlOAecAKVa0HakTkQyAf99d/c8c0xpjw0uNKuPEB99q73dULypfAikfcK/s6GDbNTVbXpW+oWwu0rAgchyvYjsddpNcB81S17Dz7rwEe9IrADwNDVPVeEUnxvjsXVwxu8TFPsSKwMSbs7N0GZUtcINSUAQLZ17oC8tAp7R4GrSoCq2oDcB+wEqgAFqhqmYg85v2V35xngM4iUoa7yL+gqhvPd8yWn5IxxoSJ9Kvgqw/BP38E/7IOxj3q5ih687vw1FB4fiJ88ls3BuEys4FgxhgTCrVb3J1B+RKoKae97gxsJLAxxnRk5wuDYdPcamddMi750BYAxhgTLs4VBld8Bf526SWtfWxLQhpjTLjoORjGPexetVtdEBysbJeF7y0AjDGmo+o5CL763XY7vK2YbIwxUcoCwBhjopQFgDHGRCkLAGOMiVIWAMYYE6UsAIwxJkpZABhjTJSyADDGmCgVVlNBiEgt8Pklfj0d2NuGzQkXdt7Rxc47urTkvK9Q1XOuphVWAdAaIlJ4vvkwIpmdd3Sx844urT1vewRkjDFRygLAGGOiVDQFwLOhbkCI2HlHFzvv6NKq846aGoAxxpi/FE13AMYYY4JYABhjTJSK+AAQkYkiskVEtovII6FuT3sSkedFpEZESoO2dReR1SKyzftvt1C2sa2JSJaIvCsi5SJSJiL3e9sj+rwBRCRRRD4VkQ3euf/E295fRD7xfudfEZGEULe1rYlIrIgUi8jr3vuIP2cAEdklIptEpERECr1tl/y7HtEBICKxwDPAJGAYcLeIDAttq9rV74GJZ217BHhbVa8C3vbeR5IG4AFVHQZcC/yL979xpJ83wEngFlUdDeQAE0XkWuAJ4OeqOhD4EviH0DWx3dwPVAS9j4ZzPuVmVc0J6v9/yb/rER0AwFhgu6ruVNU64GVgaojb1G5U9T1g/1mbpwIvev9+EZh2OdvU3lS1WlXXe/8+jLsoZBLh5w2gzhHvbbz3UuAWoMDbHnHnLiI+4A7gOe+9EOHnfAGX/Lse6QGQCVQGvfd726JJb1Wt9v69G+gdysa0JxHpB+QCnxAl5+09CikBaoDVwA7ggKo2eLtE4u/808B3gSbvfQ8i/5xPUWCViBSJyHxv2yX/rtui8FFEVVVEIrLfr4h0BhYC/6aqh9wfhU4kn7eqNgI5ItIVWAwMCW2L2peITAZqVLVIRMaFuDmhcIOqBkSkF7BaRDYHf3ixv+uRfgcQALKC3vu8bdFkj4j0BfD+WxPi9rQ5EYnHXfz/R1UXeZsj/ryDqeoB4F3gOqCriJz64y7SfuevB6aIyC7cI91bgF8Q2ed8mqoGvP/W4AJ/LK34XY/0AFgHXOX1EEgA5gLLQtymy20Z8Hfev/8OWBrCtrQ57/nvfwEVqvpU0EcRfd4AItLT+8sfEUkCJuBqIO8Cs7zdIurcVfVRVfWpaj/c/5/fUdWvE8HnfIqIpIhI6ql/A18DSmnF73rEjwQWkdtxzwxjgedV9T9C26L2IyIvAeNwU8TuAX4ELAEWANm4qbTnqOrZheKwJSI3AO8DmzjzTPjfcXWAiD1vABEZhSv6xeL+mFugqo+JyADcX8fdgWLgHlU9GbqWtg/vEdCDqjo5Gs7ZO8fF3ts44E+q+h8i0oNL/F2P+AAwxhhzbpH+CMgYY8x5WAAYY0yUsgAwxpgoZQFgjDFRygLAGGOilAWAMcZEKQsAY4yJUv8/0sLclSxPsdwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAA6l0lEQVR4nO3dd3gV5fLA8e+kEUroASQJhl6kE4r0qoAKKiJFbFelCAKK9drxehVFBBVFBBEVAcWG0kSkSQ/SO4QWeu+QNr8/9uAvFwMEkpxNcubzPHnM7tndzOrxzNl9950RVcUYY4zv8XM7AGOMMe6wBGCMMT7KEoAxxvgoSwDGGOOjLAEYY4yPCnA7gGtRuHBhjYyMdDsMY4zJUpYvX35YVUMvXZ+lEkBkZCTR0dFuh2GMMVmKiOxMab3dAjLGGB9lCcAYY3yUJQBjjPFRWWoMwBjjm+Lj44mNjeX8+fNuh5KpBQcHEx4eTmBgYKq2twRgjMn0YmNjCQkJITIyEhFxO5xMSVU5cuQIsbGxlCxZMlX72C0gY0ymd/78eQoVKmQf/lcgIhQqVOiarpIsARhjsgT78L+6a/135BMJ4KtFO/hzy2G3wzDGmEwl2yeA+MQkxi3Zxf2fL2HwjE0kJCa5HZIxJgvKkyfPP9YNGTKESpUqUbVqVVq0aMHOnSnOt+Knn35i/fr11/w3J0+ezNtvv33N+6VWtk8Agf5+/Ph4A+6tFcFHs7fS5bPF7D1+zu2wjDHZQI0aNYiOjmb16tXcc889PPvssylud6UEkJCQcNnjt2vXjueffz5dYk1Jtk8AADmD/Bl0T1WGda7O+r0nafvBfGZtOOB2WMaYLK5Zs2bkypULgHr16hEbG/uPbRYuXMjkyZN55plnqF69Otu2baNp06b079+fqKgohg0bxi+//ELdunWpUaMGLVu25MAB5/Ppiy++oE+fPgA89NBD9O3bl/r161OqVCkmTZqU5vh96jHQ9tXDqBqen97j/uKRsdE82rAkz7auQFCAT+RBY7KF139Zx/q9J9P1mJWK5+XVO25K0zFGjx5NmzZt/rG+fv36tGvXjttvv5177rnn7/VxcXF/1zY7duwYixcvRkQYNWoU77zzDu+9994/jrVv3z7+/PNPNm7cSLt27f7neNfDpxIAQMnCufnh8fq8NXUDo/7czrIdR/mwS01KFMrldmjGmCzq66+/Jjo6mrlz56Z6n06dOv39e2xsLJ06dWLfvn3ExcVd9jn+O++8Ez8/PypVqvT3VUJa+FwCAAgO9Of19pW5uXQhnp20mts+mM9/767CHdWKux2aMeYq0vpNPb39/vvvvPnmm8ydO5ccOXIA8OKLLzJlyhQAVq5cmeJ+uXPn/vv3J554gqeeeop27doxZ84cXnvttRT3uXh8cCZ+pZVP3/toXfkGpvZrRNmieXhi/Aqe/3415+IS3Q7LGJNFrFixgh49ejB58mSKFCny9/o333yTlStX/v3hHxISwqlTpy57nBMnThAWFgbA2LFjMzTm5Hw6AQCEF8jFxB4307tZaSZG7+aOj/5k4/70vb9ojMn6zp49S3h4+N8/Q4YM4ZlnnuH06dN07NiR6tWr065duxT37dy5M++++y41atRg27Zt/3j9tddeo2PHjtSqVYvChQtn9Kn8TdLjMsJboqKiNCMbwvy55TD9J67k1Pl4Xr69EvfVLWGzD43JBDZs2EDFihXdDiNLSOnflYgsV9WoS7f1+SuA5BqWLcy0fo2oW6oQL/20ll5f/8Xxs3Fuh2WMMRnCEsAlQkNy8MVDtXmxbUVmbTxA22HzWbr9qNthGWNMuktVAhCR1iKySUS2ishlp6WJSAcRURGJ8iwHicgYEVkjIqtEpGmybWt51m8VkQ8kE91r8fMTHmtciu971ScowI/OIxfx/szNVkbCGJOtXDUBiIg/MBxoA1QCuohIpRS2CwH6AUuSrX4MQFWrAK2A90Tk4t/8xPN6Wc9P6+s/jYxRNTw/v/ZtxJ01whg2awtdP1vCHisjYYzJJlJzBVAH2KqqMaoaB0wA2qew3RvAICB5MepKwB8AqnoQOA5EicgNQF5VXazOKPSXwJ3XexIZKU+OAIbcW533O1Vj3d4TtBk6j2lr9rkdljHGpFlqEkAYsDvZcqxn3d9EpCYQoapTLtl3FdBORAJEpCRQC4jw7J+8aMY/jpns2N1FJFpEog8dOpSKcDPGXTXCmdqvESUL56bXuL94/vvVnI27fBEnY4zJ7NI8COy5pTMEGJDCy5/jfLhHA0OBhcA1zbRS1ZGqGqWqUaGhoWmMNm1uLJSbSb3q83hTZ87A7R/8ydo9J1yNyRjjHW6UgwZnJvHUqVOva9+rSU0C2IPzrf2icM+6i0KAysAcEdkB1AMmi0iUqiao6pOqWl1V2wP5gc2e/cOvcMxMK9Dfj2dbV2Dco3U5G5fIXR8v4LN5MSQlZZ35FMaY9JEe5aCvxu0EsAwoKyIlRSQI6AxMvviiqp5Q1cKqGqmqkcBioJ2qRotILhHJDSAirYAEVV2vqvuAkyJSz/P0zwPAz+l8bhmqfmlnzkDzCkV4c+oGHhyzlIMnU9+L0xiT9V1vOeht27bRunVratWqRaNGjdi4cSMA3333HZUrV6ZatWo0btyYuLg4XnnlFSZOnEj16tWZOHFiusZ/1WJwqpogIn2AGYA/8LmqrhORgUC0qk6+wu5FgBkikoTzDf/+ZK89DnwB5ASmeX6ylAK5gxjRrRbjl+5m4K/raD1sPoM6VKVVpaJuh2ZM9jXtedi/Jn2PWawKtElb561rKQfdokULRowYQdmyZVmyZAmPP/44f/zxBwMHDmTGjBmEhYVx/PhxgoKCGDhwINHR0Xz00Udpii8lqaoGqqpTgamXrHvlMts2Tfb7DqD8ZbaLxrl1lKWJCF3rlqBOyYL0m7CCx76MpmvdErx0W0VyBflksVVjfM61lIM+ffo0CxcupGPHjn+vu3DhAgANGjTgoYce4t577+Xuu+/OsHgvsk+odFKmSB5+fLwB783cxMh5MSyOOcIHnWtQOSyf26EZk72k8Zt6ervWctBJSUnkz58/xTLRI0aMYMmSJUyZMoVatWqxfPnyDI3dSkGko6AAP15oU5Fxj9Tl7AVngHjE3G02QGxMNnU95aDz5s1LyZIl+e677wCnrv+qVasA2LZtG3Xr1mXgwIGEhoaye/fuq5aSTgtLABmgfhlngLhlxaK8PW0jXUdZI3pjsrr0LAc9btw4Ro8eTbVq1bjpppv4+WfnGZhnnnmGKlWqULlyZerXr0+1atVo1qwZ69evz5BBYN8oB/1LfygQCfX7gp/3cp6q8l10LK/9so4AP+E/d1WhnXUdM+aaWTno1LNy0MklxsO5o/D7q/BVezi512t/WkS4t3YE0/o1onSRPPQdv4InJ67k5Pl4r8VgjDGXk/0TgH8gdBwL7T6E2Gj4pD5s+MWrIdxYKDff9biZJ1uWY/KqvbQZOp8lMUe8GoMxxlwq+ycAABGo+QD0mA/5b4SJ3eCXfhB3xmshBPj70a9lWSb1vJlAf6HzZ4sZNH0jcQlWYtqY1MhKt6vdcq3/jnwjAVxUuAw8MhMa9IPlY+HTJrBvlVdDqFGiAFP6NqJTVASfzNnGncMXsPlAxozwG5NdBAcHc+TIEUsCV6CqHDlyhODg4FTv4xuDwCmJmQs/9oAzh6HFy3DzE14dIAaYuf4Az3+/mlMXEniudQUerh+Jn1+m6YtjTKYRHx9PbGws589buZUrCQ4OJjw8nMDAwP9Zf7lBYN9NAABnj8LkJ2Djr1CyMdw5AvKlWJU6wxw6dYEXfljN7xsOUr90IQZ3rEbx/Dm9GoMxJnvz3aeAriRXQej0tWeAeLkzQLzuR6+GEBqSg88eiOLtu6uwcvdxbh06j59X7rFLXWNMhvPtBAD/P0Dccz4UKg3fPQQ/PQ4XvHdfXkToXKcE0/o1omyRPPSbsJI+41dw7Eyc12IwxvgeSwAXFSoN/5oBjZ+BVeNhREPYvdSrIdxYKDff9riZZ24tz2/r9nPL0HnM3nTQqzEYY3yHJYDk/AOh+Uvw0FRISoLPb4U/3nQmk3lJgL8fvZuV4afeDSiYK4iHxyzj3z+u4cwFaz9pjElflgBScuPN0GsBVO0M896B0bfA4a1eDeGm4vn4uU8DejQuxfilu2gzbD7RO456NQZjTPZmCeBygvPCXZ84s4iPxsCnjSD6c/Di4GxwoD8vtK3IxO43k6TKvZ8u4q1pG7iQcE1tlY0xJkWWAK7mpjvh8UUQURd+fRLGd4bT3r0vX6dkQab3b8y9URF8OjeGdh8usGb0xpg0S1UCEJHWIrJJRLaKyPNX2K6DiKiIRHmWA0VkrIisEZENIvJCsm2fFJF1IrJWRMaLSOqnr3lb3uLQ7QdoPQi2zYaP68GGX70aQp4cAbzdoSpjHqrNsbNx3Dl8AR/O2kJCopWSMMZcn6smABHxB4YDbYBKQBcRqZTCdiFAP2BJstUdgRyqWgWoBfQQkUgRCQP6AlGqWhmn13DntJ5MhvLzg3o9occ8yBsGE++Dn3rD+ZNeDaNZhSL89mRj2la5gfdmbqbDJwvZevC0V2MwxmQPqbkCqANsVdUYVY0DJgDtU9juDWAQkHyutgK5RSQAp/l7HHDxEzMAyOl5LRfgvTrNaVGkAjw6Cxo9Dau+gRENYMcCr4aQP1cQH3SpwfCuNdl19Cy3fTCfUfNjrPOYMeaapCYBhAG7ky3Hetb9TURqAhGqOuWSfScBZ4B9wC5gsKoeVdU9wGDPun3ACVX9LaU/LiLdRSRaRKIPHTqUmnPKeAFBTv2gh6eD+MMXt8HMVyDhglfDuK3qDcx4sjGNyhbmP1M20HnkYnYe8V6FU2NM1pbmQWAR8QOGAANSeLkOkAgUB0oCA0SklIgUwLmKKOl5LbeIdEvp+Ko6UlWjVDUqNDQ0reGmrxJ1oeefUOtBWDAMRjaD/Wu8GkKRkGA+eyCKwR2rsWH/SVoPnc9Xi3bY1YAx5qpSkwD2ABHJlsM96y4KASoDc0RkB1APmOwZCO4KTFfVeFU9CCwAooCWwHZVPaSq8cAPQP20nowrcuSBO4ZB12/hzCEnCcwfAknee1RTRLinVji/PdmY2iUL8vLP6+g2egmxx856LQZjTNaTmgSwDCgrIiVFJAhnsHbyxRdV9YSqFlbVSFWNBBYD7VQ1GucWT3MAEcmNkxw2etbXE5FcIiJAC2BDOp6X95W7FR5fDBXawqzXYUwbZ/6AF92QLydjH67NW3dXYdXu47QeOp/xS3dZYTljTIqumgBUNQHoA8zA+ZD+VlXXichAEWl3ld2HA3lEZB1OIhmjqqtVdQnO+MBfwBpPHCPTcB6ZQ+5CzsSxu0fBoY3wSUNYNtqrk8dEhC51SjC9f2OqhOXjhR/W8OCYZew9fs5rMRhjsgbf7geQkU7sgZ8fh5g5ULoFtP/ImU/gRUlJyrglO3lr2kb8RXjp9orcGxWBc9FljPEV1g/A2/KFQbcfoe1g2LXImTy2aqJXrwb8/IT7b45ker/G3BSWl+e+X8NDY5ax74RdDRhjLAFkLD8/qPOY86RQaAX4sbvTkP60dx9nLVEoF988Wo/X293E0u1HuWXIPL5dttvGBozxcZYAvKFQaXh4GrQaCFt+g4/rwvrJV98vHfn5CQ/Wj2R6/0ZULJ6XZ79fbWMDxvg4SwDe4ucPDfo5pSTyhcO398P3jzp9ib3oxkK5mfCYczUQveMot7w/z54UMsZHWQLwtiIVnVISTV9w+g9/XA82TfdqCBevBmYke1Lo/tFL2X3U5g0Y40ssAbjBPxCaPu8kglyFYXwnpw/xueNeDSOiYC7GPVqX/9xZmRW7jtF66DybRWyMD7EE4Kbi1aH7bE9huQnwSX3Y+rtXQ/DzE7rVu5EZTzam5o0FePnndXT5bDE7DltNIWOyO0sAbgvI4RSWe3QmBOWBrzvA5L5eLzMdXiAXX/6rDoM6VGH9vpO0HjaPUfNjSLSrAWOyLUsAmUVYLWeAuH5fWPGVczWw7Q+vhiAidKpdgplPNqFhGafCaIdPFrLlwCmvxmGM8Q5LAJlJYDDc8gb86zcICIav7oJf+nn9aqBYPqfC6LDO1dl55Ay3ffAnH87aQrx1HzMmW7EEkBlF1Iae852rgb++dO1qoH31MGY+1YRWNxXlvZmbafeR9SI2JjuxBJBZBeb859WAC2MDhfPkYHjXmozoVovDpy/QfvgC3p62kfPx3it3bYzJGJYAMruLVwMN+jljAx/Xgy3efVIIoHXlYvz+ZBPuqRnOiLnbaDtsPku3e3cSmzEmfVkCyAoCczplJB7xPCk0roNn3sAxr4aRL1cgg+6pyteP1CU+KYl7P13Eyz+t5fSFBK/GYYxJH5YAspLwKOdJoYZPOfMGhteDTdO8HkbDsoWZ0b8xjzQsyddLdtJqyFz+2HjA63EYY9LGEkBWExgMLV+Fx2ZBrkIwvjN8/5jXawrlCgrg5dsr8UOv+oQEB/CvL6LpO34Fh09f8GocxpjrZwkgqypeA7rPgSbPwbofYHgdWPeT18OoUaIAvz7RiKdalWP62v20HDKX75fHWnE5Y7KAVCUAEWktIptEZKuIPH+F7TqIiHoawiMigSIyVkTWiMgGEXkh2bb5RWSSiGz0vHZz2k/HxwQEQbN/O4kgb3H47kGYeD+c8u7tmKAAP/q2KMuUvg0pHZqHAd+t4oHPrbicMZndVROAiPjj9PZtA1QCuohIpRS2CwH6AUuSre4I5FDVKkAtoIeIRHpeGwZMV9UKQDWyelN4NxWrAo/+AS1ehc0znKuBleO92n0MoGzREL7rcTMD29/EXzuPccv7TjmJBJtAZkymlJorgDrAVlWNUdU4YALQPoXt3gAGAeeTrVMgt4gEADmBOOCkiOQDGgOjAVQ1TlWPX/dZGPAPgEZPebqPlYefesK4jnAi1qth+PkJD9wcycynmtCgTCH+M2UDd328kHV7bQKZMZlNahJAGLA72XKsZ93fRKQmEKGqUy7ZdxJwBtgH7AIGq+pRoCRwCBgjIitEZJSI5E7pj4tIdxGJFpHoQ4e820oxSwot53Qfaz0Idi5wnhRaNgqSvPstvHj+nHz2QBTDu9Zk34nztPtoAW9N28C5OJtAZkxmkeZBYBHxA4YAA1J4uQ6QCBTH+dAfICKlgACgJvCJqtbASRIpji2o6khVjVLVqNDQ0LSG6xv8/KFeT3h8EYTXgikD4Ivb4PBWr4YhItxW9QZmPdWEjrXC+XRuDLcOncefWw57NQ5jTMpSkwD2ABHJlsM96y4KASoDc0RkB1APmOwZCO6Kc58/XlUPAguAKJyriFhVvTheMAknIZj0VCAS7v8J2g+Hg+ucmkLzh0BivFfDyJcrkLc7VGVC93oE+AndRi/hqYkrOWKPjBrjqtQkgGVAWREpKSJBQGfg747mqnpCVQuraqSqRgKLgXaqGo1z26c5gOcWTz1go6ruB3aLSHnPYVoA69PrpEwyIlCjG/ReCuVuhVmvw2fNYd8qr4dSr1QhpvZrRN/mZfhl9V5aDpnLJHtk1BjXXDUBqGoC0AeYgfOkzrequk5EBopIu6vsPhzIIyLrcBLJGFVd7XntCWCciKwGqgP/vc5zMKkRUgw6fQX3fgmn9sPIZjDzVYg/59UwggP9eeqW8kzt24jSoXl4+rtV3DdqCdutA5kxXidZ6dtXVFSURkdHux1G1nfuGPz2slNcrmApuGMYlGzs9TCSkpTxy3bx9rSNXEhI4olmZejRpDRBATY/0Zj0JCLLVTXq0vX2f5ovylkA2n8ED0x25gqMvQN+7uP14nJ+fsJ9dW9k1lNNaFXJ6TnQ9gOrMmqMt1gC8GWlmkCvhU6p6ZXfwPC6sP5nr08gK5I3mOFdazLmodqcj0/k3k8X8dyk1Rw/G+fVOIzxNZYAfF1QLqfU9GN/QJ6i8O0DMOE+OLHn6vums2YVijDzySb0bFKaSX/F0uK9ufzwlw0SG5NRLAEYR/Hq8NhsJxls+8O5Glj6GSR5d+JWziB/nm9TgV+faEiJQrl46ltnkDjm0GmvxmGML7BBYPNPR7fDr09CzGwIrw13fABF/1H+KcMlJSnfLN3FoOkbuRCfRK+mpenVtDTBgf5ej8WYrMwGgU3qFSwJ9/8Id30KR7bBp41g1hsQf/7q+6YjPz+hW70bmTWgCW2qFGPYrC20HjqP+VusJIgx6cESgEmZCFTrDH2iofI9MH+wM5M4Zq7XQykSEsywzjX4+pG6iAj3j15K3/ErOHjKuwnJmOzGEoC5styF4O5PnSsCTYIv28GPveDMEa+H0rBsYab1a0T/lmWZvnY/Ld6by5eLdpCYlHVuYxqTmdgYgEm9+HMw9x1Y+AHkyAu3/te5ShDxeigxh07zys/r+HPrYaqE5ePNuypTNTy/1+MwJiu43BiAJQBz7Q6sg1/6Q+xSZwbx7UOhUGmvh6Gq/LJ6H2/8up7Dpy/Qre6NPH1refLlDPR6LMZkZpYATPpKSoLlY+D31yDhAjR+2plQFpDD66GcPB/PkN828+WiHRTMHcSLt1XkzuphiAtXJsZkRpYATMY4tR+mv+A0pi9UFu4YCpENXQll7Z4TvPjTWlbtPk69UgV5o31lyhYNcSUWYzITewzUZIyQYtBxDNz3PSTGOY1nfnrclUHiymH5+LFXfd68qzIb9p2izbD5vDVtA2cuJHg9FmOyArsCMOkn7izMewcWfugMEt/yBlS/z5VB4iOnLzBo+ka+jY7lhnzBvHJ7JVpXLma3hYxPsltAxnsOrIdf+8PuJXBjA7htCBSp4Eooy3ce5cUf17Jx/ykalwvl9XY3UbJwiu2njcm2LAEY70pKcvoNzHwF4k5D/b7Q+Bmn+JyXJSQm8eWinQyZuZm4hCR6NCnF403LkDPISkoY32AJwLjjzGGn+cyqbyB/CWj7HpS7xZVQDp48z3+nbuCnlXsJy5+TV+6oxC2VitptIZPtpWkQWERai8gmEdkqIs9fYbsOIqKehvCISKCIjBWRNSKyQUReuGR7fxFZISK/XusJmSwid2G46xN4aAoE5IRvOsLEbq6Umy6SN5ihnWswoXs98uQIoMdXy3n4i2XssHaUxkddNQGIiD9Ob982QCWgi4j8ozSkiIQA/YAlyVZ3BHKoahWgFtBDRCKTvd4Pp8+wye4iG0LPP6H5y7BlJnxU2xksToz3eij1ShXi174Nefn2SkTvOMYt789j8IxNnIvzbulrY9yWmiuAOsBWVY1R1ThgAtA+he3eAAYBySt0KZBbRAKAnEAccBJARMKB24BR1x++yVICgpwJY72XQMlG8NtL8Glj2LnI66EE+vvxSMOS/DGgCbdVvYGPZm+l5ZC5TF+7zxrQGJ+RmgQQBuxOthzrWfc3EakJRKjqlEv2nQScAfYBu4DBqnqx4etQ4Fkg6Up/XES6i0i0iEQfOmRlgLOFApHQdSJ0/gYunIIxreGn3s54gZcVyRvM+52qM7F7PUKCA+j59V888PlSth60BjQm+0vzRDAR8QOGAANSeLkOkAgUB0oCA0SklIjcDhxU1eVXO76qjlTVKFWNCg0NTWu4JjOpcJtzNdDwSVg9AT6sBdGfe70LGUDdUoX49YmGvN7uJlbuPk7rofN4a+oGTtskMpONpSYB7AEiki2He9ZdFAJUBuaIyA6gHjDZMxDcFZiuqvGqehBYAEQBDYB2nu0nAM1F5Os0novJioJyQ8vXoOcCKFbF6UQ2qiXs+cvroQT4+/Fg/UhmP92UDjXD+XReDM0Hz+GnFXvstpDJllKTAJYBZUWkpIgEAZ2ByRdfVNUTqlpYVSNVNRJYDLRT1Wic2z7NAUQkN05y2KiqL6hquGf7zsAfqtotPU/MZDFFKsCDv8Ddo+DkHvisuZMMzh69+r7prHCeHAy6pyo/Pl6fYvmC6T9xJR1HLGLtnhNej8WYjHTVBKCqCUAfYAbOEzvfquo6ERkoIu2usvtwII+IrMNJJGNUdXVagzbZlAhU7Qh9lkG9XrB8LHwUBSu+diaWeVmNEgX46fEGDOpQhe2Hz3DHR3/y4o9rOHYmzuuxGJMRbCKYybz2r4EpT8PuxRBeB24bDDdUcyWUE+fiGfr7Zr5ctJM8OQJ4+pZydKlTggB/q6doMj+bCWyypqQkWDXeKSlx7ihEPQLNX4ScBVwJZ/OBU7w2eR0Ltx2hQrEQXmt3E/VKFXIlFmNSyxKAydrOHYfZb8KyUZCzoDNwXP0+8PP+N3BVZdra/bw5ZQN7jp/jtqo38O+2FQnLn9PrsRiTGpYATPawbzVMfdqpNBpeG9oOhuLVXQnlfHwin86N4ZO5WwHo2aQ0PRqXtiJzJtOxBGCyj6QkZ97AzFecyWNRDzslJnIVdCWcPcfP8d+pG5iyeh9h+XPy77YVaVvFeg+YzMMSgMl+zh2HOW/D0pEQnBdavAI1HwQ/d76BL445wuu/rGfDvpPUKVmQV++oxE3F87kSizHJWQIw2deBdTD1Wdj5p/OUUNvBEFHHlVASk5QJy3bx3m+bOXY2js61IxhwS3kK58nhSjzGgCUAk92pwtrvnQJzp/ZBta7OQHFIUVfCOXEung9mbWHswh3kDPSnb4uyPFg/kqAAe2zUeJ8lAOMbLpyGee/CouEQmBOaPAd1e4B/oCvhbD14mjenrGf2pkOULJybF9tWpEXFIjY+YLzKEoDxLUe2wfTnYctvULgctBkEpZu7Fs7sTQd549f1xBw6Q6OyhXnptkqULxbiWjzGt1gCML5p8wwnERyNgQq3w61vOuWoXRCfmMTXi3cy9PctnDofT9e6JXiqVXkK5g5yJR7jOywBGN+VcMG5JTRvMCQlQP0noNFTTiVSFxw7E8fQ3zfz9ZJd5Aryp1+Lsjxws40PmIxjCcCYE3vg99dgzbcQUhxaDYQq9zhF6Fyw5cAp3piygXmbnfGBF9pUoJU1qTcZwBKAMRftWgzTnoN9KyGiHrR5G4rXcC2c2ZsO8uaUDWw9eJqbSxXipdsr2vwBk64sARiTXFISrBwHs153ZhPX6OZMJMtTxJVw4hOT+GbJLt7/fTMnzsVzb60IBtxajiIhwa7EY7IXSwDGpOT8CZj7DiwZAQE5ockzULcnBLgzcevE2Xg+/GMLYxftIMjfj8ebleGRhiUJDrT6Qub6WQIw5koOb4XfXoTN06FASedpofJtXRsf2H74DG9N3cBv6w9QPF8wz7QuT/tqYfj52fiAuXaWAIxJja2/w/R/w+FNUKop3PoWFK3kWjiLY47wnynrWbvnJFXD8/HSbZWoU9Kdoncm67pcAkjVc2ci0lpENonIVhF5/grbdRAR9TSER0QCRWSsiKwRkQ0i8oJnfYSIzBaR9SKyTkT6Xe+JGZOuyrSEXgugzTuwdyWMaABTBsCZI66EU69UISb3bsiQe6tx6NQF7v10ET2/Ws6Ow2dcicdkL1e9AhARf2Az0AqIxent20VV11+yXQgwBQgC+qhqtIh0xWkQ31lEcgHrgabABeAGVf3Ls99y4M5Lj3kpuwIwXnX2KMx5C5aNhqA80PQ5qP0YBLgzcetcXCKj5sfwydxtxCcm0a3ejfRtXpYCNpHMXEVargDqAFtVNUZV44AJQPsUtnsDGAScT7ZOgdwiEgDkBOKAk6q6T1X/AlDVUzjN5sOu5YSMyXC5CkLbd6HXQoioDTP+DR/Xg41TneJzXpYzyJ8nWpRlzjNNuadWBGMX7qDxu7MZOW8bFxISvR6PyfpSkwDCgN3JlmO55MNaRGoCEao65ZJ9JwFngH3ALmCwqh69ZN9IoAawJKU/LiLdRSRaRKIPHTqUinCNSWdFKkC37+G+SeAXABO6wJftYf9ad8IJCeatu6swvX9jom4swH+nbqTFe3P5ZdVestKYnnFfmueei4gfMAQYkMLLdYBEoDhQEhggIqWS7ZsH+B7or6onUzq+qo5U1ShVjQoNDU1ruMZcv7KtPOMD78L+1fBpI5jcF04dcCWcckVDGPNwHb5+pC4hwYE8MX4Fd368kKXbj159Z2NIXQLYA0QkWw73rLsoBKgMzBGRHUA9YLJnILgrMF1V41X1ILAA+HuAGOfDf5yq/pDWEzHGK/wDoW53eOIvqNvLmUz2YU2Y/x7En3MlpIZlC/PrEw15956qHDhxnns/XcRjX0az7dBpV+IxWUdqBoEDcAaBW+B88C8DuqrqustsPwd42jMI/BxQQVUfFpHcnn07A2uAscBRVe2f2mBtENhkOke2wW8vw6YpkC/CaUJTuYNr8wfOxSXy+YLtfDJnG+fiE+lSJ4J+LcoRGmIdyXzZdQ8Cq2oC0AeYgTNY+62qrhORgSLS7iq7DwfyiMg6nA//Maq6GmgA3A80F5GVnp+213hOxrivUGno8g08+AvkzA/fPwKjWsKuFIe0MlzOIH96NyvDnGeacl/dEkxYupum787mg1lbOBuX4EpMJvOyiWDGpJekRFg1Hma9Aaf3Q6U7nSuCgiVdCynm0GkGTd/IjHUHKBKSg/4ty3FvVDgB/lZ62pfYTGBjvOXCaVj4ISz8wOk/UKc7NH4achZwLaToHUd5a9pGlu88RunQ3DzfpiItrTWlz7AEYIy3ndwLf7zpDBTnzA9Nnoeof7k2kUxVmbHuAO9M30jM4TPUjizAC20rUrOEe4nJeIclAGPcsm81/PYSbJ8LBUtDq9ed9pQuffuOT0xi4rLdDP19C4dPX6BN5WI8fWt5SofmcSUek/EsARjjJlWnQf3MV+DQRqcRza1vQvg//p/0mjMXEvhsfgyfzYvhfEISnWpH0L9FWYrktR4E2Y0lAGMyg8QEWPEVzP4vnDkIN90NLV91rVE9wKFTF/jojy2MW7KLQH8/Hm1Uku6NSxESHOhaTCZ9WQIwJjO5cAoWfOAMFmuiM1DcaIBTf8glOw6fYfBvm/h19T4K5AqkT/OydKtXghwB1owmq7MEYExmdHIvzH4TVoyD4LzQ6GknGQS6dxtmdexxBk3fyIKtRwjLn5OnWpXjzhph+FszmizLEoAxmdmBdTDzVdg6E/KVgBYvQ+V7wM+95/XnbznEoOkbWbvnJOWLhvBs6/I0r2CPjmZFlgCMyQpi5jilJfavhhuqQauBTmcylyQlKVPX7mPwjE3sOHKW2pEFeK51BaIirStZVmIJwJisIikJ1k6CWQPhxG4o3cJ5dLRYFddCuvjo6LBZWzh06gItKxbh6VvLU6FYXtdiMqlnCcCYrCb+PCz7DOYNhvMnoGonaP4i5C/hWkhn4xL4YuEOPpmzjdMXErizehhPtixHiUK5XIvJXJ0lAGOyqnPH4M/3YfEIQDPFE0PHz8YxYm4MYxZsJ0mVrnVK0Kd5Was6mklZAjAmqzsR68wfWPkN5MgLjZ6Euj0hMKdrIR04eZ4PZm1hwrLdBPn78a+GkXRvXJp8OW0OQWZiCcCY7OLAOvj9ddgyA0KKQ7MXoFpX8A9wLaTth8/w/szNTF61l7zBAfRsWpqH6keSK8i9mMz/swRgTHazYwH8/irELoPC5Z0ZxeXbulZjCGD93pO899smZm08SOE8OXiieRm61ClBUICVn3aTJQBjsiNV2Pirc0VwZAtE1HV6ENxY39Wwlu88yjvTN7Fk+1HCC+SkX4uy3FUjzPoQuMQSgDHZ2cUaQ3PedprRlL0VWrwCxSq7FpKqMn/LYd6dsYk1e05QKjQ3T7UqR9vKN+Bns4q96rpbQnp2bi0im0Rkq4g8f4XtOoiIehrCIyKBIjJWRNaIyAYReeFaj2mMSQX/AIh6GPqucK4Adi+GEQ3hh+5wbIcrIYkIjcuFMrlPA0Z0q0WAn9DnmxXc9uGfzNpwgKz05TO7Sk1TeH+cpvCtgFic3r5dVHX9JduFAFOAIKCPpyl8V6CdqnYWkVzAeqApsDs1x7yUXQEYk0rnjsGfQ2HJCKdVZdS/nK5keYq4FlJikvLLqr0MmbmZXUfPUqNEfp65pTz1yxR2LSZfkZYrgDrAVlWNUdU4YALQPoXt3gAGAeeTrVMgt4gEADmBOODkNRzTGHM9chZwZg/3XQE17oNlo2BYdfjjP86kMhf4+wl31ghj1oAm/PeuKuw/cZ6uo5bQZeRionccdSUmX5eaBBCG8439oljPur+JSE0gQlWnXLLvJOAMsA/YBQxW1aOpOWayY3cXkWgRiT506FAqwjXG/C1vcbhjGPReCuVugXnvwrBqTinq+HOuhBTo70fXuiWY/XRTXr2jElsOnuaeEYt4aMxS1sS6k5x8VZqH5EXEDxgCDEjh5TpAIlAcKAkMEJFS13J8VR2pqlGqGhUaGprWcI3xTYXLQMcvoPtcCKsFM1+GD2rC8i+cAWQXBAf683CDksx7tinPt6nAyt3HueOjP+n+ZTQb9590JSZfk5oEsAeISLYc7ll3UQhQGZgjIjuAesBkz0BwV2C6qsar6kFgARCVimMaYzJC8erQ7Xt4aArkC4df+sHwOrBmklOEzgW5ggLo2aQ0859txpMty7Fo2xFaD51P72/+YuvBU67E5CtSMwgcgDNg2wLnQ3oZ0FVV111m+znA055B4OeACqr6sIjk9uzbGWcwONXHvMgGgY1JR6qweTrMegMOroOilaH5y1DuVlcnkx0/G8dn82MYs2AH5+MTaV89jL4tylKycG7XYsrqrnsQWFUTgD7ADGAD8K2qrhORgSLS7iq7DwfyiMg6nA/5Maq6+nLHvLZTMsakiQiUbwM9/4QOoyH+LIzvBKNvge3zXQsrf64gnrm1AvOfbcZjjUoxbe0+Wg6ZyzPfrWL30bOuxZUd2UQwY4wjMR5WjoM5g+DUXqcRTfOXIfwfXxy96uCp83wyZxvjluwiKUnpGBVO72ZlCC9gJahTy2YCG2NSJ/48RI+G+UPg7GEo1waav+TqrGKA/SfO8/GcrUxYuhtFuTcqgt7NylA8v3vVULMKSwDGmGtz4TQs+QQWfAgXTsBNd0Ozf0Phsq6Gtff4OYbP3sq30bsRhM51Ini8aRmK5Qt2Na7MzBKAMeb6nDsGCz+CxZ9Awjmo1gWaPAsFIl0NK/bYWYbP3sp30bH4+Qld65SgV9PSFM1rieBSlgCMMWlz+pDTmWzZKNBEqPkANHoa8qU4h9Nrdh89y0d/bGXSX7H4WyJIkSUAY0z6OLkX5r8Hy8eC+Dl1hho+CSFFXQ1r15GzfDR7C9//tYcAP6Fr3RL0alKaIpYILAEYY9LZsZ1OaYmV34B/ENTtDvX7Qe5Croa168hZPvxjCz+scBJBF7sisARgjMkgR7Y5fQjWfAdBuZ0+xTf3drVpPcDOI2f48I+t/Lhiz9+3hno2Ke2Tg8WWAIwxGevgRpj7Nqz70Wlaf3NvqNcLgvO5GtbOI2cYPnsrP/y1Bz8/oXPtCHo1Lc0N+Xzn8VFLAMYY79i/Fua85bSqDM4H9Z9wrgpyhLga1u6jzlNDk5bH4idCp9oR9GxamjAfmEdgCcAY4117Vzq3hjZPc/oT1H8C6nTPFIng4znbmLTcqUh/T60IHm9amoiC2XdmsSUAY4w79ix3EsGW3yBnQWjQF2o/BjnyuBvW8XOMmLONict2k6TK3TXD6N2sDDcWyn5F5ywBGGPcFRvt3Bra+jvkKgQN+kHtR52BYxftP3GeEXO3MX7pLhKSlPbVi9O7WRlKh7qboNKTJQBjTOawe6mTCLb9AbkKexLBI64ngoMnz/PpvBjGLdnJhYQkbq9anD7NylC+mLu3rNKDJQBjTOaya7FzayhmdqZKBIdPX2DU/O18tWgHZ+ISaX1TMfo0L0PlMHefZkoLSwDGmMwpkyaCY2fiGLNgO2MW7uDU+QRaVChCn+ZlqFGigKtxXQ9LAMaYzO3SRFD/CWeMwOXB4hPn4vly4Q5GL9jO8bPxNCxTmD7Ny1C3ZEHExc5p18ISgDEma9i1BOYOgm2znKeG6vfJFI+PnrmQwLglOxk5bzuHT1+gdmQBejcrQ5NyoZk+EaQpAYhIa2AY4A+MUtW3L7NdB2ASUNvTE/g+4Jlkm1QFaqrqShHpAvwbUGAv0E1VD18pDksAxviQ3ctg3juex0cLQL3eTr0hl2cWn49P5Nvo3YyYs429J85TJSwfvZuV4ZZKRfHzy5yJ4LoTgIj44zRwbwXE4vT27aKq6y/ZLgSYAgQBfVQ1+pLXqwA/qWppT6P5vUAlVT0sIu8AZ1X1tSvFYgnAGB+0ZznMfdeZUBacD+r2gno9naTgoriEJH5cEcvHc7ax88hZyhXNw+NNy3B71RsI8L9qu3Wvuu6m8EAdYKuqxqhqHDABaJ/Cdm8Ag4DzlzlOF8++AOL5yS3OtVNenIRgjDH/K6wWdJ0A3edCZCOn3tD7VWDWQDhzxLWwggL86FS7BLOeasKwztUB6D9xJc3fm8s3S3ZxISHRtdhSKzUJIAzYnWw51rPubyJSE4hQ1SlXOE4nYDyAqsYDvYA1eK4EgNEp7SQi3UUkWkSiDx06lIpwjTHZUvHq0Hkc9FwAZVs6PYuHVoHfXoLTB10LK8Dfj/bVw5jerzEj769FgVyB/PvHNTR+Zzaj5sdwNi7BtdiuJs3XKSLiBwwBBlxhm7o4t3jWepYDcRJADaA4sBp4IaV9VXWkqkapalRoaGhawzXGZHXFKkPHL6D3EqhwGywa7iSCac/BiT2uheXnJ9xyUzF+6t2Arx6pQ2Sh3PxnygYavP0HH8zawomz8a7FdjmpSQB7gIhky+GedReFAJWBOSKyA6gHTBaR5PebOuP59u9RHUBVt6kzCPEtUP9agzfG+LDQ8tDhM+gTDZXvcVpVflAdfukPx3a4FpaI0KhsKBN73Mz3vW6mZokCDJm5mfpvz+KtqRs4ePJyd8m9LzWDwAE4g8AtcD74lwFdVXXdZbafAzx9cRDYc4WwG2ikqjGedcWB5UBVVT0kIm8AuVT1slcRYIPAxpgrOLYTFgyDFV9BUiJU7QSNnoLCZd2OjA37TvLJnG38unovAf5+dKwVTo/GpSlRyDsVSNP6GGhbYCjOY6Cfq+qbIjIQiFbVyZdsO4f/TQBNgbdVtd4l2/UE+gHxwE7gIVW94oiOJQBjzFWd3AsLP4ToMZBwHm6602leX6yy25Gx4/AZPp0Xw/fLY0lIcuoN9Wpamoo35M3Qv2sTwYwxvuX0IVj0ESwbDXGnoFwbaPw0hP/jc9DrDpw8z+g/tzNu8U7OxCXSvEIRejUtTe3IjGmjaQnAGOObzh2DJSNhySfO7yWbQONnILIhuDyD98TZeL5ctIMxC3dw9EwcUTcWoFfT0jQrXyRdJ5VZAjDG+LYLpyH6c+eq4PQBCK8DjQZAuVtdTwRn4xKYuGw3o+ZvZ8/xc5QvGkLPpqW4vWpxAtNhUpklAGOMAYg/7wwUL/gATuyCopWdweJKd4Kfv7uhJSbxy6q9jJi7jc0HThOWPyePNSpJp9olyBl0/bFZAjDGmOQS42Ht986EssOboGApaPgkVO0MAUGuhpaUpMzedJBP5mwjeucxCuQKZHr/xhTNG3xdx7MEYIwxKUlKgo2/wvz3YN9KCCnuVCCt+aDrpagBlu04yu8bDvBCm4rXfQxLAMYYcyWqTpvKP9+HHfOdYnN1ekDdHpArY57O8Za0FIMzxpjsTwTKtICHfoVHfocSN3sKz1WG6f925hdkM5YAjDHmUhG1oct46LUIKt4OS0bA0Krwc284vMXt6NKNJQBjjLmcopXg7pHQ9y+o9RCsmQQf1YaJ3Zw+BVmcJQBjjLmaApFw22Dov9aZO7B9HnzWHMbe4YwbZKGx1OQsARhjTGrlCYUWLzuJoNUbcGgzfHUXjGziPFKamHlr/6fEEoAxxlyr4LzQoC/0Xw13fABxZ2HSv+CjWrD0M4g/53aEqWIJwBhjrldADqj1IPReCp2+hlyFYerTzpNDc9+Fs0fdjvCKLAEYY0xa+flBxTvg0d/hoakQVhNm/8fzCOkLcHz31Y/hggC3AzDGmGxDBCIbOD8H1jl9CZaOhCWfQuUOzm2jYlXcjvJvdgVgjDEZoehNcNcI6LcK6vWCTVNhRENn0DhmTqZ4csgSgDHGZKR84XDrm/DkWmjxCuxfC1+2h08bO/MKXHxyKFUJQERai8gmEdkqIs9fYbsOIqIXG8KLyH0isjLZT5KIVPe8FiQiI0Vks4hsFJEO6XJGxhiTGeUs4Mwh6L8G2n3otKv8/hGnkf2ij51+BV6Wmqbw/jhN4VsBsThN4buo6vpLtgsBpgBBQJ+LPYGTvV4F+ElVS3uWXwf8VfUlT+P4gqp6+EqxWDE4Y0y2kZQEW2Y4fQl2LYTgfBD1iFN8LqRYuv6ptBSDqwNsVdUYVY0DJgDtU9juDWAQcP4yx+ni2feifwFvAahq0tU+/I0xJlvx84PybeBf0+DRWU6rygVDYWgV+Kk3HNyQ8SGkYpswIPkzTLGedX8TkZpAhKpOucJxOgHjPdvn96x7Q0T+EpHvRKRoSjuJSHcRiRaR6EOHDqUiXGOMyWLCo6DTV/DEcqfm0Lof4ON68PU9GTpgnOZBYM/tmyHAgCtsUxc4q6prPasCgHBgoarWBBYBg1PaV1VHqmqUqkaFhoamNVxjjMm8CpaCtu/Ck+ug+Uuwb5VnwLgRnNqf7n8uNQlgDxCRbDncs+6iEKAyMEdEdgD1gMkXB4I9OuP59u9xBDgL/OBZ/g6oeU2RG2NMdpWrIDR+5v8HjPPfCLmLpPufSU0CWAaUFZGSIhKE82E++eKLqnpCVQuraqSqRgKLgXYXB4E9Vwj3kuz+vzojz78ATT2rWgD/M6hsjDE+LzAYaj4Ancc5Ywbp7KozgVU1QUT6ADMAf+BzVV0nIgOBaFWdfOUj0BjYraoxl6x/DvhKRIYCh4CHrzl6Y4wx1816AhtjTDZnPYGNMcb8D0sAxhjjoywBGGOMj7IEYIwxPsoSgDHG+ChLAMYY46Oy1GOgInII2HmduxcGfLHgnJ23b7Hz9i2pPe8bVfUftXSyVAJICxGJTuk52OzOztu32Hn7lrSet90CMsYYH2UJwBhjfJQvJYCRbgfgEjtv32Ln7VvSdN4+MwZgjDHmf/nSFYAxxphkLAEYY4yPyvYJQERai8gmEdkqIs+7HU9GEpHPReSgiKxNtq6giMwUkS2efxZwM8aMICIRIjJbRNaLyDoR6edZn63PXUSCRWSpiKzynPfrnvUlRWSJ5z0/0dPIKdsREX8RWSEiv3qWs/15i8gOEVkjIitF5GLTret+n2frBCAi/sBwoA1QCegiIpXcjSpDfQG0vmTd88AsVS0LzPIsZzcJwABVrYTTkrS3579zdj/3C0BzVa0GVAdai0g9YBDwvqqWAY4Bj7gXYobqB2xItuwr591MVasne/7/ut/n2ToBAHWAraoao6pxOG0p27scU4ZR1XnA0UtWtwfGen4fC9zpzZi8QVX3qepfnt9P4XwohJHNz10dpz2LgZ4fBZoDkzzrs915A4hIOHAbMMqzLPjAeV/Gdb/Ps3sCCAN2J1uO9azzJUVVdZ/n9/1AUTeDyWgiEgnUAJbgA+fuuQ2yEjgIzAS2AcdVNcGzSXZ9zw8FngWSPMuF8I3zVuA3EVkuIt096677fX7VnsAm+1BVFZFs+9yviOQBvgf6q+pJ50uhI7ueu6omAtVFJD/wI1DB3YgynojcDhxU1eUi0tTlcLytoaruEZEiwEwR2Zj8xWt9n2f3K4A9QESy5XDPOl9yQERuAPD886DL8WQIEQnE+fAfp6o/eFb7xLkDqOpxYDZwM5BfRC5+ucuO7/kGQDsR2YFzW7c5MIzsf96o6h7PPw/iJPw6pOF9nt0TwDKgrOfpgCCgMzDZ5Zi8bTLwoOf3B4GfXYwlQ3ju/44GNqjqkGQvZetzF5FQzzd/RCQn0Apn/GM2cI9ns2x33qr6gqqGq2okzv/Tf6jqfWTz8xaR3CIScvF34BZgLWl4n2f7mcAi0hbnfqE/8LmqvuluRBlHRMYDTXFKxB4AXgV+Ar4FSuCU0r5XVS8dKM7SRKQhMB9Yw//fE/43zjhAtj13EamKM+jnj/Nl7ltVHSgipXC+GRcEVgDdVPWCe5FmHM8toKdV9fbsft6e8/vRsxgAfKOqb4pIIa7zfZ7tE4AxxpiUZfdbQMYYYy7DEoAxxvgoSwDGGOOjLAEYY4yPsgRgjDE+yhKAMcb4KEsAxhjjo/4PcPkcloOT1MIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "os.environ[\"KMP_DUPLICATE_LIB_OK\"] = \"TRUE\"\n",
    "\n",
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 生成数据\n",
    "X_train = torch.randn(1000, 30)\n",
    "y_train = torch.sin(X_train) + torch.randn(1000, 30) * 0.1\n",
    "X_test = torch.randn(100, 30)\n",
    "y_test = torch.sin(X_test) + torch.randn(100, 30) * 0.1\n",
    "\n",
    "# 假设我们有一个包含两个隐藏层的神经网络\n",
    "model = torch.nn.Sequential(\n",
    "    torch.nn.Linear(30, 20),\n",
    "    torch.nn.ReLU(),\n",
    "    torch.nn.Linear(20, 10),\n",
    "    torch.nn.ReLU(),\n",
    "    torch.nn.Linear(10, 30)\n",
    ")\n",
    "\n",
    "# 定义损失函数和优化器\n",
    "loss_fn = torch.nn.MSELoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "# 不使用 L2 正则化的情况\n",
    "train_losses = []\n",
    "test_losses = []\n",
    "\n",
    "for epoch in range(50):\n",
    "    # 计算训练损失\n",
    "    y_pred = model(X_train)\n",
    "    loss = loss_fn(y_pred, y_train)\n",
    "    train_losses.append(loss.item())\n",
    "    \n",
    "    # 使用优化器更新权重\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 计算测试损失\n",
    "    with torch.no_grad():\n",
    "        y_pred = model(X_test)\n",
    "        loss = loss_fn(y_pred, y_test)\n",
    "        test_losses.append(loss.item())\n",
    "    \n",
    "# 绘制训练损失和测试损失的曲线\n",
    "plt.plot(train_losses, label='train')\n",
    "plt.plot(test_losses, label='test')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "# 使用 L2 正则化的情况\n",
    "model = torch.nn.Sequential(\n",
    "torch.nn.Linear(30, 20),\n",
    "torch.nn.ReLU(),\n",
    "torch.nn.Linear(20, 10),\n",
    "torch.nn.ReLU(),\n",
    "torch.nn.Linear(10, 30)\n",
    ")\n",
    "\n",
    "#定义损失函数和优化器\n",
    "loss_fn = torch.nn.MSELoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.1) # 加入 L2 正则化\n",
    "\n",
    "train_losses = []\n",
    "test_losses = []\n",
    "\n",
    "for epoch in range(50):\n",
    "    # 计算训练损失\n",
    "    y_pred = model(X_train)\n",
    "    loss = loss_fn(y_pred, y_train)\n",
    "    train_losses.append(loss.item())\n",
    "\n",
    "    # 使用优化器更新权重\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 计算测试损失\n",
    "    with torch.no_grad():\n",
    "        y_pred = model(X_test)\n",
    "        loss = loss_fn(y_pred, y_test)\n",
    "        test_losses.append(loss.item())\n",
    "\n",
    "#绘制训练损失和测试损失的曲线\n",
    "plt.plot(train_losses, label='L2-train')\n",
    "plt.plot(test_losses, label='L2-test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这个例子中，使用了 PyTorch 的数据生成函数 `torch.randn()` 和 `torch.sin()` 来生成训练数据和测试数据。然后构建了一个包含两个隐藏层的神经网络，并使用随机生成的数据进行训练。在第一个循环中，我们没有使用 L2 正则化；在第二个循环中，我们使用了 L2 正则化。\n",
    "\n",
    "在两个循环中，我们都计算了训练数据和测试数据的损失，并将训练数据和测试数据的损失添加到列表 train_losses 和 test_losses 中，然后使用 Matplotlib 绘制了两组损失的曲线。在使用 L2 正则化的情况下，你可以看到训练损失和测试损失的差距变小了，这表明 L2 正则化有助于减小过拟合。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Next 5-6 Dropout](./5-6%20Dropout.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
